Skip to content

feat(annotate): carry agent-facing element context on HTML and live-app pinpoints - #1517

Merged
backnotprop merged 1 commit into
mainfrom
feat/html-element-context
Sep 12, 2026
Merged

backnotprop merged 1 commit into
mainfrom
feat/html-element-context

Conversation

@backnotprop

Copy link
Copy Markdown
Owner

Problem

Pinpointing an element in a raw-HTML or live-app annotate session exported either the bridge's placeholder quote (Feedback on: "[element: Navigation]") or the element's flattened textContent. The bridge already built a unique selector, tag and text snapshot, then nothing in the export read them. An agent reading the feedback could not tell which element the comment was about. Investigation and design: DESIGN_html-annotation-context.md / .html (untracked, repo root).

What ships

Capture (bridge, buildElementContext), at click time, since only the bridge can see the DOM and an SPA may have replaced the element by submit:

field source cap
tag id classes the element; generated/hashed classes skipped 32 / 100 / 8 (+"+N more")
path ancestors, positional only when a same-tag sibling reads identically 8 segments / 512
role name explicit or implicit ARIA role; aria-label → aria-labelledby → alt → title → <label for> → own short text 32 / 120
attrs allowlist only (href src alt title type name role placeholder for target rel aria-* data-testid/test/cy/qa/component/id/annotate); absolute URLs lose query+fragment, data: keeps its media type, javascript: dropped 10 × 120
text innerText, whitespace-collapsed, word-boundary cut 300
outline opening tag with id/classes/attrs, then children as bare tags; adaptive: two levels, then one, then a per-tag count, whichever first fits 600
children rect landmark heading component count; viewport box; nearest region; nearest preceding h1–h6; nearest data-component/data-testid ancestor (no React fiber reads) —
page live app only: route + title 2048 / 200

Hard cap 2 KiB serialized per primary, 1 KiB per shift-click extra target, shedding outline → text → attrs → classes → path → heading → landmark → component. Never captured: form values, on*, style, script/style/template contents, full innerHTML, viewer overlays. A 400-row list costs the same bytes as a chip (pinned by test).

Trust boundary (parseHtmlElementContext): re-collapses every scalar (newlines → markdown structure), re-applies every cap and the allowlist, drops unknown keys, defuses 3+ backtick runs in the outline, re-enforces the byte bound. A malformed context is dropped, never fatal to the selection.

Export (elementContextExportBlock): under the comment, a 4-backtick html fence of the skeleton plus selector / path / role · name · component / attrs / text / box / near lines. A text-less pinpoint's placeholder heading becomes Feedback on the <nav> element — "Primary"; a pinpoint with real quoted text keeps its quote line and gains the block. Extra targets list selector · path on their existing one-liner. Grouped live-app export omits the route (the ## Page: heading carries it); the single-entry form includes it.

## 1. Feedback on the <nav> element — "Primary"
> this

````html
<nav id="site-nav" class="site-nav sticky" aria-label="Primary" data-component="AppNav">
  <ul class="nav-list">
    <li>Home</li>
    <li>About</li>
  </ul>
  <button class="nav-toggle"></button>
</nav>
  • selector nav#site-nav
  • path body > div#root > header.site-header > nav#site-nav
  • role navigation · name "Primary" · component data-component=AppNav
  • attrs aria-label="Primary" data-component="AppNav"
  • box 0,0 1280×64 (viewport 1280×800)
  • near header.site-header · heading h1 "Acme Analytics"

**Copy**: `exportAnnotationEntry` renders one annotation (no number, route included) and backs a new per-row Copy button in the annotation panel, shown only on cards that carry an element; other surfaces render byte-identically.

**Archive**: `elementTag` / `elementSelector` / `elementPath` / `elementRole` / `elementName` / `pageUrl`, identity only (additive per the cross-tool field contract; auto-vendored to Pi).

## Compatibility

- Annotations without `elementContext` export byte-identically (pinned by the existing byte-exact test and a new one).
- `HtmlElementAnchor` untouched; restore never reads the context. Share links drop it like anchors (pinned).
- No `BRIDGE_PROTOCOL_VERSION` bump: additive in both directions.
- `@plannotator/ui` public type `Annotation` gains an optional field; no host seam changes.

## Not in this PR

- `packages/core/html-anchor.ts` (`buildPersistedHtmlAnchor` / `projectHostThreads`, the host persistence helpers) does not carry `elementContext` yet; a host persisting through it drops the field. Follow-up with its own byte-budget ladder.
- Drag text selections on HTML surfaces carry no context (pinpoints only), per the open question.

## Test plan

- [x] `bun x tsc --noEmit -p packages/ui/tsconfig.json`, `packages/shared`
- [x] `DOM_TESTS=1 bun test packages/ui packages/editor packages/review-editor` (only the pre-existing order-dependent DocBadges / artifactDocument failures, which pass in isolation)
- [x] `bun test packages/shared packages/core`
- New tests: bridge capture on a `<nav>` (identity, path, role, name, allowlist, skeleton, surroundings; script text / handlers / style / secrets absent; URL scrubbing), 400-child container under budget, extra-target context; boundary parser (newline collapse, allowlist, unknown keys, caps + shedding, malformed → dropped without rejecting the selection); export shape, heading rule, byte-identical without context, route only on the single entry, copied entry ⊂ full export, fence/inline-code escape; archive identity fields; share links drop it; panel Copy button presence and clipboard content.

…pp pinpoints

A pinpointed element exported as a one-word placeholder ('[element:
Navigation]') or its flattened textContent. The bridge now captures a
bounded description at click time (tag, id, classes, ancestor path, role,
accessible name, allowlisted attributes, rendered text, an adaptive HTML
skeleton, box, landmark, heading, component hint, live route) as
elementContext; the parent re-validates and re-caps it; the export prints
a fenced skeleton plus selector/path/role/name lines under the comment;
the annotation panel gains a per-row Copy for element-bearing cards; the
feedback archive records element identity. Additive: annotations without
the field export byte-identically, share links drop it, no protocol bump.
@backnotprop
backnotprop merged commit 782d974 into main Sep 12, 2026
28 checks passed
@backnotprop
backnotprop deleted the feat/html-element-context branch September 12, 2026 23:20
backnotprop added a commit that referenced this pull request Sep 12, 2026
Element context on HTML and live-app pinpoints (#1517, #1520). Core stays
0.25.2; peer ranges unchanged from 0.38.2.
backnotprop added a commit that referenced this pull request Sep 12, 2026
* fix(ui): remove the annotation panel's per-row Copy button

The panel's card chrome goes back to Edit and Delete. The element context
still rides every HTML and live-app pinpoint into the exported feedback;
exportAnnotationEntry stays as a pure helper for hosts and tests.

* chore(ui): bump @plannotator/ui to 0.39.0

Element context on HTML and live-app pinpoints (#1517, #1520). Core stays
0.25.2; peer ranges unchanged from 0.38.2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant